Face-Tracking Method with High Accuracy

ABSTRACT

A face-tracking method with high accuracy is provided. The face-tracking method includes generating an initial face shape according to the detected face region of an input image and a learned data base, wherein the initial face shape comprises an initial inner shape and an initial outer shape; generating a refined inner shape by refining the initial inner shape according to the input image and the learned data base; and generating a refined outer shape by searching an edge of the refined outer shape from the initial outer shape toward the limit of outer shape.

FIELD OF THE INVENTION

The present invention relates to computer vision, and more particularly,to a face-tracking method with high accuracy.

BACKGROUND OF THE INVENTION

Generally speaking, face-tracking refers to a computer vision technologythat extracts the shapes of human faces in arbitrary digital images. Itdetects facial features and ignores anything else in surrounding, suchas furniture or dogs. According to the related art, there are manyconventional face tracking methods (e.g., snake, AAM, CLM . . . , etc.)based on face detection to detect face region and then set an initialshape (which is composed by feature points) inside the region, and thecontent of a given part in face region of an image is extracted to getfeatures and then go fine tuning the face shape to fit features in theimage face.

However, these methods may result in false shape extractions due toover/under face region detection or target-like background noises, andthe following processes (e.g., the power saving application or thecamera application) based on the face detection results would beaffected by the false shape extractions. Therefore, there is a need foran innovative face-tracking scheme which is capable of extracting faceshapes accurately.

SUMMARY OF THE INVENTION

The present invention provides a face-tracking method with highaccuracy. The face-tracking method comprises generating an initial faceshape according to a detected rectangle face region in an input imageand a learned data base, wherein the initial face shape comprises aninitial inner shape and an initial outer shape; generating a refinedinner shape by refining the initial inner shape according to thefeatures in the input image and the learned data base; and generating arefined outer shape by searching features composed by edges of therefined outer shape from the initial outer shape outward to limits ofthe defined possible outer shape.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating a face-tracking method with highaccuracy according to an exemplary embodiment of the present invention.

FIG. 2( a) is a diagram illustrating the face region of face detectionresult and initial inner shape of the face-tracking method with highaccuracy according to an exemplary embodiment of the present invention.

FIG. 2( b) is a diagram illustrating the refined inner shape of theface-tracking method with high accuracy according to an exemplaryembodiment of the present invention.

FIG. 3 is a diagram illustrating searching edges of the refined outershape from the initial outer shape toward the limit of outer shape ofthe face-tracking method with high accuracy according to an exemplaryembodiment of the present invention.

FIG. 4 is a diagram illustrating a direct searching process of theface-tracking method with high accuracy according to an exemplaryembodiment of the present invention.

FIG. 5 is a diagram illustrating a 2D directional searching process ofthe face-tracking method with high accuracy according to anotherexemplary embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The main concept of the present invention is to improve theface-tracking accuracy especially when the background of an image iscomplicated or messy. In such condition, the performance of conventionalface-tracking methods will be deteriorated, especially at face border.However, the present invention proposes a face-tracking method foranalyzing images (i.e., face shape border detection) from the inside ofa face toward outside with a predetermined distance so as to avoid thebackground noise interference and improve the face-tracking accuracy.

Please refer to FIG. 1 and FIG. 2. FIG. 1 is a flowchart illustrating aface-tracking method 100 with high accuracy according to an exemplaryembodiment of the present invention. FIG. 2( a) is a diagramillustrating the face region of face detection result and initial innershape of the face-tracking method with high accuracy according to anexemplary embodiment of the present invention. FIG. 2( b) is a diagramillustrating the refined inner shape of the face-tracking method withhigh accuracy according to an exemplary embodiment of the presentinvention. Provided that substantially the same result is achieved, theSteps in FIG. 1 need not be in the exact order shown and need not becontiguous, that is, other Steps can be intermediate. Besides, someSteps in FIG. 1 may be omitted according to various types of embodimentsor requirements. In this embodiment, the detailed operation forface-tracking may comprise the following Steps:

Step101: Receive an input image 202 (Refer to FIG. 2);Step 102: Generate an initial face shape according to the detected faceregion (the rectangle with dot line in FIG. 2( a)) of the input image202 and a learned data base, wherein the initial face shape comprises atleast an initial inner shape 204 and an initial outer shape (not shownhere, defined in FIG. 3 as ‘initial shape’);Step 103: Generate a refined inner shape 208 by refining the initialinner shape 204 in accordance with at least the input image 202 and thelearned data base;Step 104: Generate a refined outer shape by searching an edge of therefined outer shape from the initial outer shape toward the limit ofouter shape.

In Step 101, the input image 202 can be a frame of a video, a picture,and so on. After the input image 202 is received in Step101, Step 102 isexecuted to operate a face detection on the input image 202 to get aface region (the rectangle with dot line in FIG. 2( a)) for generatingan initial inner shape 204 and an initial outer shape according to alearned data base. Please refer to FIG. 2( a), which is a diagramillustrating the face detection of the face-tracking method 100 withhigh accuracy according to an exemplary embodiment of the presentinvention. An initial face shape at a detected face region may be setwith help of the learned data base of many various “human faces”. Morespecifically, in Step 102, a generic face shape model with n featurepoints is built according to the learned data base, and a generativeshape of the initial face shape S_(i)(θ)=1, . . . ,n , can be describedas equation (1):

S _(i)(θ)=sR( s _(i) +Γ_(i)γ)+t; θ={s,R,γ,t}  (1);

wherein the initial face shape comprises at least the initial innershape 204 and the initial outer shape, and (s_(i), Γ_(i)) are thelearned data base which comprises average shape s_(i) - , anddeformation span basics Γ₁, and θ comprises geometric factors s, R, t,and Γ, wherein s represents a scaling factor, R represents a rotationmatrix (composed by roll, yaw and pitch), t represents a translationfactor, and Γ represents deformation parameters which are adjustableparameters utilized for fitting various face shape in the input image202.

It should be noted that in Step 102, the geometric factors s, R, t, andΓ are just rough values and under no posture conditions, e.g. roll, yaw,or pitch. In other words, these geometric factors are not refined yet,and have to be refined through further fitting process. Consequently,the initial inner shape 204 and the initial outer shape are just roughresults as well and need to be refined at the following Step 103.Moreover, the n feature points may have errors due to the differencebetween the generic face shape model of the learned data base and thereal face shape in the input image 202.

In Step 103, some optimization algorithms are used to iteratively tunethe initial shape to match the extracted features in real image faceshape, and do not stop until some criteria are met. For betterunderstanding of technical features of the present invention, oneoptimization algorithm to match an image (x_(i), y_(i)) with model asmentioned above to find optimum θ and z_(i) are described in equation(2):

$\begin{matrix}{\min\limits_{{\{ z_{i}\}}_{i = 1}^{n},\theta}{\sum\limits_{i = 1}^{n}{{\rho \left( {{{\left\lbrack {x_{i},{y_{i};z_{i}}} \right\rbrack - {S_{i}(\theta)}}}^{2};\sigma} \right)}.}}} & (2)\end{matrix}$

However, this is for illustrative purposes only, and is not meant to bea limitation of the present invention. The optimization algorithms orschemes may be modified according to different optimization algorithmsor schemes. As a person skilled in the art can readily understanddetails of the optimization methods described in equation (2), andfurther description is omitted here for brevity.

Please refer to FIG. 2( b), which is a diagram illustrating the refinedinner shape of the face-tracking method 100 with high accuracy accordingto an exemplary embodiment of the present invention. The refined innershape 208 thereby is generated through the optimization algorithmsmentioned above, and the refined inner shape 208 may be supposed to bemore accurate than the initial inner shape 204 as shown in FIG. 2( a).For example, the position and the shape of the mouth in the refinedinner shape 208 are presented more accurately comparing with the initialinner shape 204. Furthermore, the geometric factors which describes aposture of the refined inner shape 208 is also refined in Step 103,wherein the geometric factors comprises at least the scaling factor s,the rotation matrix R, and the translation factor t as mentioned above.In other words, the refined inner shape 208 comprises the posture, e.g.roll, yaw, or pitch information in Step 103.

In general, since there is only skin around the inner face shape, andthe background around the outer face shape may appear with someunexpected objects, the background around the outer face shape of theinput image 202 is much more complicated than the background around theinner face shape. Considering the fact mentioned above, the presentinvention processes the refined inner shape 208 first, and thenprocesses the refined outer shape or the whole face shape. In this way,the refined inner shape 208 would be generated stably and precisely. InStep 104, after inner shape has been extracted, an initial outer shapeand many scan line segments may be set for searching correct outer shape(face border) from inside to outside direction from a face center.

Please refer to FIG. 3, which is a diagram illustrating searching anedge of the refined outer shape from the initial outer shape toward thelimit of outer shape according to an exemplary embodiment of the presentinvention. In FIG. 3, the initial outer face shape has 17 feature pointsand may be separated into several groups. For example, the 17 featurepoints may be separated into 4 groups, and more specifically, each ofthe groups has a common base point 302 which are selected from thepoints of the refined inner shape 208. Then, 17 scan lines 304 whichcorresponding to the 17 feature points are generated from thecorresponding common base points 302 toward the limit of outer faceshape respectively, and 17 line segments 306 are set respectively on the17 scan lines to indicate a searching range for the following searchingprocess.

Any object outside the line segments 306 would be ignored, hence thesearching process may avoid identifying most of the undesired objects atbackground and the searching process would be efficient. For example,line segments can be defined as a n dimensions 2D image point p_(i)(xi,yi) arrays ArrayP_(k)[p_(i)], i=0,1, . . . ,n-1 and k =0,1, . . . ,16.Please note that this setting is initially under no posture (no roll,yaw and pitch) condition. The image coordinates of each point in thescan line segment arrays ArrayP_(k)[p_(i)] should be transformed tocorrect position before doing searching operation. For example, eachpoint p_(i) in ArrayP_(k)[p_(i)] may be transformed to (as equation(3)):

p′ _(i) =sRp _(i) +t   (3);

wherein s, R, t are the scaling factor, the Rotation matrix (composed byhead roll, yaw and pitch) and the translation factor as defined inequation (1). Therefore, refined arrays of the scan line segment 306 forsearching usage is ArrayP_(k)[p′_(i)]. According to the embodiment ofthe present invention, the searching process may be configured to adirect searching in the one dimensional refined arraysArrayP_(k)[p′_(i)] of the scan line segment 306 in an in-to-outdirection.

Firstly, the input image 202 is processed by any one well known edgedetection method to get an ‘edge map’ for further processing, forexample, the Sobel edge detection. However, this is for illustrativepurposes only, and is not meant to be a limitation of the presentinvention. The edge detection method or scheme may be modified, whereinother methods may be employed according to different edge detection.Please refer to FIG. 4, which is a diagram illustrating a directsearching process according to an embodiment of the present invention.An edge map 402 is generated from the input image 202 according to theaforementioned edge detection method or scheme, Then, refined arraysArrayP_(k)[p′_(i)] of the scan line segment 306 are searched from indexi=0 to i=n−1 (i.e. from inward to outward direction) to find a maximumedge point, which is the border of the outer face shape we find in thisStep, assume it is p_(max(i,k)), where (i, k) represents the maximumedge point at indexed number i in k scan line segment.

In another embodiment of the present invention, a searching process isconfigured to be 2D directional searching along the refined arraysArrayP_(k)[p′_(i)] of the scan line segment 306 with a 2D patch ‘edgedetector’. Pease refer to FIG. 5, which is a diagram illustrating a 2Ddirectional searching process according to another embodiment of thepresent invention. A 2D patch 502 as shown in FIG. 5 is capable ofperforming a directional searching for the maximum edge point. The patchis a window (for example, 15×15 pixels, but not limited) centering at apoint in the refined arrays ArrayP_(k)[p′_(i)] of the scan line segment306 and is suitable for implementing a 2D edge detector. For betterunderstanding of technical features of the present invention, one 2Dedge detection method to search along the refined arraysArrayP_(k)[p′_(i)] of the scan line segment 306 are described as anexample. However, this is for illustrative purposes only, and is notmeant to be a limitation of the present invention. Many well know 2Dedge detection methods or schemes can be used to achieve the sameobjective also belong to the scope of the present invention. As result,the maximum edge point p_(max(i,k)) can be found as the border of theouter face shape.

After each maximum edge point p_(max(i,k)) has been found (if no edgefound in some line segment, the maximum edge point in p_(max(i,k)) ofthe line segment would be omitted), one optimization algorithm similarto Equation (2) can be used to determine the refined outer shape.Equation (4) shows this optimization equation, where the shape (faceborder) generated from the searching process with total n maximum edgepoints (e.g., 17 points in this embodiment) is subtracted by predictedshape S_(k)(θ) from the learned data base and z_(k) and θ are estimatedby optimization and minimization process until some converge condition(stop criteria) met. Note that we only need take deformation parametersγ into account inθ (includes the scaling factor s, the rotation matrix(composed by roll, yaw and pitch) R, the translation factor t, and thedeformation parameters γ, but not limited to) because the scaling factors, the rotation matrix R, and the translation factor t are determinedwhen we obtained the refined inner shape 208. In addition, some geometryconstrains, for example, reflection symmetry with respect to the leftface and the right face, can be imposed upon the optimization andminimization process to improve the fitting correctness as Equation (5).

$\begin{matrix}{\min\limits_{{{\{ z_{k}\}}k},\theta}{\sum\limits_{k = 0}^{n - 1}{\rho \left( {{{\left\lbrack {p_{\max {({i,k})}};z_{k}} \right\rbrack - {S_{k}(\theta)}}}^{2};\sigma} \right)}}} & (4)\end{matrix}$

Assume center at (0,0,0), no rotation for(k=0; k<n12; k++)

|[p _(max(i,k)) ; z _(k) ]−[p _(max(i,n−k)) ; z _(n−k)]|<δ  (5)

Wherein |.| denotes the distance between two points, δ is a thresholdvalue. It is an advantage of the present invention that the presentinvention method can provide an improved flow for face-tracking process.In addition, the improved flow for face-tracking process is suitable forvarious kinds of, where a traditional face-tacking or detection processcan be altered with ease based upon the embodiments disclosed above, toprevent the related art problems.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A face-tracking method with high accuracy,comprising: generating an initial face shape according to a detectedface region of an input image and a learned data base, wherein theinitial face shape comprises an initial inner shape and an initial outershape; generating a refined inner shape by refining the initial innershape according to the input image and the learned data base; andgenerating a refined outer shape by searching an edge of the refinedouter shape from the initial outer shape toward the limit of outershape.
 2. The face-tracking method of claim 1, wherein the step ofgenerating the refined inner shape further comprises: generating therefined inner shape and the plurality of geometric factors whichdescribes a posture of the refined inner shape.
 3. The face-trackingmethod of claim 2, wherein the plurality of geometric factors comprisesa scaling factor, a rotation matrix, and a translation factor.
 4. Theface-tracking method of claim 1, wherein the step of generating therefined outer shape by searching a border of the refined outer shapefrom the initial outer shape toward the limit of outer shape comprises:setting at least one scan line segment which starts from a point of theinitial outer shape; searching an at least one maximum edge point alongthe at least one scan line segment; and generating the refined outershape by optimizing the at least one maximum edge point.
 5. Theface-tracking method of claim 4, wherein the step of setting at leastone scan line segment which starts from a point of the initial outershape comprises: setting at least one angle for the at least one scanline segment respectively; and setting at least one length for the atleast one scan line segment respectively.
 6. The face-tracking method ofclaim 5, wherein setting the at least one angle for the at least onescan line segment respectively is based on the refined inner shape andlearned data base
 7. The face-tracking method of claim 5, whereinsetting the at least one length for the at least one scan line segmentrespectively is based on the refined inner shape and learned data base.8. The face-tracking method of claim 5, wherein the step of setting theat least one scan line segment which starts from the point of theinitial outer shape further comprises: transforming the at least onescan line segment in accordance with the plurality of geometric factorswhich describes the posture of the refined inner shape.
 9. Theface-tracking method of claim 8, wherein the plurality of geometricfactors comprises a scaling factor, a rotation matrix, and a translationfactor.
 10. The face-tracking method of claim 4, wherein the step ofsearching the at least one maximum edge point along the at least onescan line segment comprises: performing an edge detection process uponthe input image to find edges in the input image; and finding the atleast one maximum edge point by performing an one dimensional edgedetection process along the at least one scan line segment.
 11. Theface-tracking method of claim 10, wherein the edge detection process isa Sobel edge detection process.
 12. The face-tracking method of claim 4,wherein the step of searching the at least one maximum edge point alongthe at least one scan line segment further comprises: finding the atleast one maximum edge point by performing a two dimensional edgedetection process along the at least one scan line segment.
 13. Theface-tracking method of claim 4, wherein the step of generating therefined outer shape by optimizing the at least one maximum edge pointcomprises: optimizing the at least one maximum edge point in accordancewith the initial shape and the plurality geometric factors.
 14. Theface-tracking method of claim 13, wherein the step of generating therefined outer shape by optimizing the at least one maximum edge pointfurther comprises: optimizing the at least one maximum edge pointaccording to a deformation factor and a symmetric reflection constraint.